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Abstract — Given the diversity of commercial Cloud services, 
performance evaluations of candidate services would be crucial 
and beneficial for both service customers (e.g. cost-beneflt analy- 
sis) and providers (e.g. direction of service improvement). Before 
an evaluation implementation, the selection of suitable factors 
(also called parameters or variables) plays a prerequisite role in 
designing evaluation experiments. However, there seems a lack 
of systematic approaches to factor selection for Cloud services 
performance evaluation. In other words, evaluators randomly 
and intuitively concerned experimental factors in most of the 
existing evaluation studies. Based on our previous taxonomy 
and modeling work, this paper proposes a factor framework for 
experimental design for performance evaluation of commercial 
Cloud services. This framework capsules the state-of-the-practice 
of performance evaluation factors that people currently take 
into account in the Cloud Computing domain, and in turn can 
help facilitate designing new experiments for evaluating Cloud 
services. 

Index Terms — Cloud Computing; commercial Cloud services; 
performance evaluation; experimental design; factor framework 

I. Introduction 

Along with the boom in Cloud Computing, an increasing 
number of commercial providers have started to offer public 
Cloud services [24|, |30|. Since different commercial Cloud 
services may be supplied with different terminologies, defini- 
tions, and goals f30l, performance evaluation of those services 
would be crucial and beneficial for both service customers 
(e.g. cost-benefit analysis) and providers (e.g. direction of 
improvement) (24). Before implementing performance eval- 
uation, a proper set of experiments must be designed, while 
the relevant factors that may influence performance play a 
prerequisite role in designing evaluation experiments |18|. In 
general, one experiment should take into account more than 
one factor related to both the service to be evaluated and the 
workload. 

After exploring the existing studies of Cloud services per- 
formance evaluation, however, we found that there was a lack 
of systematic approaches to factor selection for experimental 
design. In most cases, evaluators identified factors either ran- 
domly or intuitively, and thus prepared evaluation experiments 
through an ad hoc way. For example, when it comes to 



the performance evaluation of Amazon EC2, different studies 
casually considered different EC2 instance factors in different 
experiments, such as VM type f3T\, number ["321, geographical 
location [ 16J, operation system (OS) brand |24|, and even CPU 
architecture |fT6l and brand 1271 . etc. In fact, to the best of our 
knowledge, none of the current Cloud performance evaluation 
studies has used "experimental factors" deliberately to design 
evaluation experiments and analyze the experimental results. 

Therefore, we decided to establish a framework of suitable 
experimental factors to facilitate applying experimental design 
techniques to the Cloud services evaluation work. Unfortu- 
nately, it is difficult to directly point out a full scope of exper- 
imental factors for evaluating performance of Cloud services, 
because the Cloud nowadays is still chaotic compared with 
traditional computing systems f33\. Consequently, we used a 
regression manner to construct this factor framework. In other 
words, we tried to isolate the de facto experimental factors 
from the state-of-the-practice of Cloud services performance 
evaluation. In fact, the establishment of this factor framework 
is a continuation of our previous work ||2TI . Il22l . ||23| that 
collected, clarified and rationalized the key concepts and their 
relationships in the existing Cloud performance evaluation 
studies. Benefitting from such a de facto factor framework, 
new evaluators can explore and refer to the existing evaluation 
concerns for designing their own experiments for performance 
evaluation of commercial Cloud services. 

Note that, as a continuation of our previous work, this study 
conventionally employed four constrains, as listed below. 

• We focused on the evaluation of only commercial Cloud 
services, rather than that of private or academic Cloud 
services, to make our effort closer to industry's needs. 

• We only investigated Performance evaluation of commer- 
cial Cloud services. The main reason is that not enough 
data of evaluating the other service features could be 
found to support the generalization work. For example, 
there are little empirical studies in Security evaluation of 
commercial Cloud services due to the lack of quantitative 
metrics ll22l . 

• We considered Infrastructure as a Service (laaS) and Plat- 



form as a Service (PaaS) without considering Software 
as a Service (SaaS). Since SaaS with special function- 
alities is not used to further build individual business 
applications ||4l, the evaluation of various SaaS instances 
could comprise an infinite and exclusive set of factors 
that would be out of the scope of this investigation. 

• We only explored empirical evaluation practices in aca- 
demic publications. There is no doubt that informal 
descriptions of Cloud services evaluation in blogs and 
technical websites can also provide highly relevant in- 
formation. However, on the one hand, it is impossible 
to explore and collect useful data from different study 
sources all at once. On the other hand, the published 
evaluation reports can be viewed as typical and peer- 
reviewed representatives of the existing ad hoc evaluation 
practices. 

The remainder of this paper is organized as follows. Section 
[n] briefly introduces the four-step methodology that we have 
used to establish this factor framework. Section III specifies 
the tree-structured factor framework branch by branch. An 
application case is employed in Section|lV]to demonstrate how 
the proposed factor framework can help facilitate experimental 
design for Cloud services performance evaluation. Conclusions 
and some future work are discussed in Section [Vl 

II. Methodology of Establishing the Framework 

As previously mentioned, this factor framework is estab- 
lished based on our previous work, which is mainly composed 
of four steps, as listed below and respectively specified in the 
following subsections: 

• Conduct a systematic literature review (SLR). 

• Construct a taxonomy based on the SLR. 

• Build a conceptual model based on the taxonomy. 

• Establish an experimental factor framework at last. 

A. Conduct Systematic Literature Review 

The foundation for establishing this factor framework is 
a systematic literature review (SLR) on evaluating commer- 
cial Cloud services}^ As the main methodology applied for 
Evidence-Based Software Engineering (EBSE) [8|, SLR has 
been widely accepted as a standard and systematic approach 
to investigation of specific research questions by identifying, 
assessing, and analyzing published primary studies. Following 
a rigorous selection process in this SLR, as illustrated in Figure 
[T] we have identified 46 Cloud services evaluation studies 
covering six commercial Cloud providers, such as Amazon, 
GoGrid, Google, IBM, Microsoft, and Rackspace, from a 
set of popular digital publication databases (all the identified 
evaluation studies have been listed online for reference: http: 
|//www.mendeley.com /groups/l 104801/slr4cloud/papers/|. The 
evaluation experiments in those identified 46 studies were 
thoroughly analyzed. In particular, the atomic experimental 
components, such as evaluation requirements. Cloud service 
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Fig. 1. The study selection sequence in the SLR on evaluating commercial 
Cloud services. 



features, metrics, benchmarks, experimental resources, and 
experimental operations, were respectively extracted and ar- 
ranged. 

B. Construct Taxonomy 

During the analysis of these identified evaluation studies, 
we found that there were frequent reporting issues ranging 
from non-standardized specifications to misleading explana- 
tions ||2T1 . Considering that those issues would inevitably 
obstruct comprehending and spoil drawing lessons from the 
existing evaluation work, we created a novel taxonomy to 
clarify and arrange the key concepts and terminology for 
Cloud services performance evaluation. The taxonomy is 
constructed along two dimensions: Performance Feature and 
Experiment. Moreover, the Performance Feature dimension is 
further split into Physical Property and Capacity parts, while 
the Experiment dimension is split into Environmental Scene 
and Operational Scene parts, as shown in Figure [2] The details 
of this taxonomy has been elaborated in II21I . 

C. Build Conceptual Model 

Since a model is an abstract summary of some concrete 
object or activity in reality |26|, the identification of real 
and concrete objects/activities plays a fundamental role in 
the corresponding modeling work. Given that the taxonomy 
had capsuled relevant key concepts and terminology, we 
further built a conceptual model of performance evaluation 
of commercial Cloud services to rationaUze different abstract- 
level classifiers and their relationships f23l. In detail, we 
used a three-layer structure to host different abstract elements 
for the performance evaluation conceptual model. To save 
space, here we only portray the most generalized part hosted 
in the top classifier layer, as shown in Figure |3] which 
reflects the most generic reality of performance evaluation 
of a computing paradigm: essentially, performance evaluation 
can be considered as exploring the capacity of particular 
computing resources with particular workloads driven by a 
set of operations. 
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Fig. 3. Conceptual model of Cloud services performance evaluation in the 
top classifier layer. 



D. Establish Factor Framework 

In fact, the specific classifiers in the abovementioned 
conceptual model 1231 has implied the state-of-the-practice 
of performance evaluation factors that people currently took 
into account in the Cloud Computing domain. According to 
different positions in the process of an evaluation experiment 
ill], the specific classifiers of Workload and Computing 
Resource indicate input process factors; the specific classifiers 
of Capacity suggest output process factors; while the 
Operation classifiers are used to adjust values of input process 
factors. The detailed experimental factors for Cloud services 
performance evaluation are elaborated in the next section. 
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Fig. 4. The workload factors for experimental design. 



III. The Tree-structured Factor Framework 

As mentioned previously, the experimental factors for per- 
formance evaluation of commercial Cloud services can be 
categorized into two input process groups (Workload and 
Computing Resource) and one output process group (Capac- 
ity). Thus, we naturally portrayed the factor framework as a 
tree with three branches. Each of the following subsections 
describes one branch of the factor tree. 

A. Workload Factors 

Based on our previous work ||2T| , ll23l . we found that 
a piece of workload used in performance evaluation could 
be described through one of three different concerns or a 
combination of them, namely Terminal, Activity, and Object. 
As such, we can adjust the workload by varying any of 
the concerns through different experimental operations. The 
individual workload factors are listed in Figure [4] 

1) Terminal: In contrast with services to be evaluated 
in the Cloud, clients and particular Cloud resource (usually 
VM instances) issuing workload activities can be viewed 
as terminals. Correspondingly, the geographical location or 
number of both clients llTl and VM instances [14J have 
been used to depict the relevant workload. Meanwhile, the 
terminal type can also be used as a workload factor For 
example, the authors evaluated Cloud network latency by using 
client and EC2 instance respectively to issue pings |2|. In this 
case, the terminal type has the equal essence to the factor 



communication scope (cf. Subsection III-Bli. 

2) Activity: The concept "activity" here describes an in- 
herent property of workload, which is different from, but 
adjustable by, experimental operations. For example, disk I/O 
request as a type of activity can be adjusted by operations like 
the number or time of the requests. In fact, the number- and 
time-related variables, such as activity duration flT], frequency 



||3, number jS], and timing ifTOll . have been widely considered 
as workload factors in practice. Furthermore, by taking a 
particular Cloud resource being evaluated as a reference, the 
factor activity direction can be depicted as input or output ||2l. 
As for the activity sequence in a workload, the arrangement 
generates either sequential [2 1 or parallel 114] activity flows. 

3) Object: In a workload for Cloud services performance 
evaluation, objects refer to the targets of the abovemen- 
tioned activities. The concrete objects can be individual mes- 
sages 1 13 1, data files lfl4l . and transactional jobs/tasks Q 
in fine grain, while they can also be coarse-grained work- 
flows or problems [6]. Therefore, the object number and 
object size/complexity are two typical workload factors in 
the existing evaluation studies. Note that we do not consider 
object location as a workload factor, because the locations 
of objects are usually hosted and determined by computing 
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resources (cf. Subsection III-B i. In particular, a workload 
may have multiple object size/complexity-related factors in 
one experiment. For example, a set of parameters of HPL 
benchmark, such as the block size and process grid size, should 
be tuned simultaneously when evaluating Amazon EC2 fS)- 

B. Computing Resource Factors 

According to the physical properties in the performance 
feature of commercial Cloud services ||2TI . the Cloud Com- 
puting resource can be consumed by one or more of four 
basic styles: Communication, Computation, Memory (Cache), 
and Storage. In particular, the VM Instance resource is an 
integration of all the four basic types of computing resources. 
Overall, the computing resource factors can be organized as 
Figure |5] shows. 

1) Communication: As explained in fl\\. Communica- 
tion becomes a special Cloud Computing resource because 
commercial Cloud services are employed inevitably through 
Internet/Ethernet. As such, the Ethernet I/O Index is usually 
pre-supplied as a service-level agreement (SLA) by service 
providers. In practice, the scope and level of communication 
have been frequently emphasized in the performance eval- 
uation studies. Therefore, we can summarize two practical 
factors: The factor Communication Scope considers intra- 
Cloud and wide-area data transferring respectively ll24ll . while 
the Communication Level distinguishes between IP-level and 
MPI-message-level networking lfT2l . 

2) Computation: When evaluating PaaS, the Computation 
resource is usually regarded as a black box |fT6ll . Whereas, 
for laaS, the practices of Computation evaluation of Cloud 
services have taken into account Core Number |f3|. Elastic 
Compute Unit (ECU) Number, Thread Number [2], and a 
set of CPU characteristics. Note that, compared to physical 
CPU core and thread, ECU is a logical concept introduced 
by Amazon, which is defined as the CPU power of a 1.0-1.2 
GHz 2007 Opteron or Xeon processor ESll . When it comes 
to CPU characteristics, the Architecture (e.g. 32 bit vs. 64 
bit GSl) and Brand (e.g. AMD Opteron vs. Intel Xeon |l27l) 
have been respectively considered in evaluation experiments. 
Processors with the same brand can be further distinguished 
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Fig. 5. The computing resource factors for experimental design. 



between different CPU Models (e.g. Intel Xeon E5430 vs. Intel 
Xeon X5550 f3l). In particular, CPU Frequency appears also 
as an SLA of Cloud computation resources. 

3) Memory (Cache): Since Memory/Cache could closely 
work with the Computation and Storage resources in com- 
puting jobs, it is hard to exactly distinguish the affect to 
performance brought by Memory/Cache. Therefore, not many 
dedicated Cloud memory/cache evaluation studies can be 
found from the literature. In addition to the SLA Memory 
Size, interestingly. Physical Location and Size of cache (e.g. 
L1=64KB vs. L2=1MB in Amazon ml.* instances |28|) have 
attracted attentions when analyzing the memory hierarchy. 
However, in 1281 . different values of these factors were ac- 
tually revealed by performance evaluation rather than used for 
experimental design. 

4) Storage: As mentioned in f2T|, Storage can be either the 
only functionality or a component functionality of a Cloud 
service, for example Amazon S3 vs. EC2. Therefore, it can 
be often seen that disk-related storage evaluation also adopted 
experimental factors of evaluating other relevant resources like 



VM instances (cf. Subsection III-B5 i. Similarly, the predefined 



Storage Size acts as an SLA, while a dedicated factor of 
evaluating Storage is the Geographical Location. Different 
geographical locations of Storage resources can result either 
from different service data centers (e.g. S3 vs. S3-Europe 
||29l) or from different storing mechanisms (e.g. local disk 
vs. remote NFS drive |31|). In addition, although not all 
of the public Cloud providers specified the definitions, the 
Storage resource has been distinguished among three types of 
offers: Blob, Table and Queue |24|. Note that different Storage 
Types correspond to different sets of data-access activities, as 
described in f22|. 

5) VM Instance: VM Instance is one of the most popular 
computing resource styles in the commercial Cloud service 
market. The widely considered factors in current VM Instance 
evaluation experiments are Geographical Location, Instance 
Number, and VM Type 13], HI, III, |[l6l, 128], l32l . 
The VM Type of a particular instance naturally reflects its 
corresponding provider, as demonstrated in |24|. Moreover, 
although not common, the OS Brand (e.g. Linux vs. Windows 
||24]) and Physical Location fl \ also emerged as experimental 
factors in some evaluation studies. Note that the physical lo- 
cation of a VM instance indicates the instance's un-virtualized 
environment, which is not controllable by evaluators in evalu- 
ation experiments [7|. In particular, recall that a VM Instance 
integrates above four basic types of computing resources. We 
can therefore find that some factors of evaluating previous 
resources were also used in the evaluation of VM Instances, 
for example the CPU Architecture and Core Number 15], |281 . 

C. Capacity Factors 

As discussed about the generic reality of performance 



evaluation in Subsection II-C it is clear that the capacities 
of a Cloud computing resource are intangible until they are 
measured. Meanwhile, the measurement has to be realized 
by using measurable and quantitative metrics |20|. Therefore, 
we can treat the values of relevant metrics as tangible repre- 
sentations of the evaluated capacities. Moreover, a particular 
capacity of a commercial Cloud service may be reflected by 
a set of relevant metrics, and each metric provides a different 
lens into the capacity as a whole For example. Benchmark 
Transactional Job Delay f25] and Benchmark Delay (19] 
are both Latency metrics: the former is from the individual 
perspective, while the latter from the global perspective. As 
such, we further regard relevant metrics as possible output 
process factors 1 1 1 when measuring a particular Cloud service 
capacity, and every single output process factor can be used 
as a candidate response [IJ in the experimental design. Since 
we have clarified seven different Cloud service capacities 121], 
i.e. Data Throughput, Latency, Transaction Speed, Availability, 
Reliability, Scalability, and Variability, the possible capacity 
factors (metrics) can be correspondingly categorized as Figure 
[6] shows. Due to the limit of space, it is impossible and unnec- 
essary to exhaustively list all the metrics in this paper In fact, 
the de facto metrics for performance evaluation of commercial 
Cloud services have been collected and summarized in our 
previous work ll22ll . 
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IV. Application of the Factor Framework 

Since the factor framework is inherited from the aforemen- 
tioned taxonomy and modeling work, it can also be used for, 
and in turn be validated by, analyzing the existing studies of 
Cloud services performance evaluation, as described in |21|, 
||23]| . To avoid duplication, we do not elaborate the analysis 
application scenario, and the corresponding validation, of 
the factor framework in this paper. Instead, we particularly 
highlight and demonstrate how this factor framework can help 
facilitate designing experiments for evaluating performance of 
commercial Cloud services. 

Suppose there is a requirement of evaluating Amazon EC2 
with respect to its disk I/O. Recall that relevant factors 
play a prerequisite role in designing evaluation experiments. 
Given the factor framework proposed in this paper, we can 
quickly and conveniently lookup and choose experimental 
factors according to the evaluation requirement. To simplify 
the demonstration, here we constrain the terminal to be clients, 
while only consider the direction of disk I/O and data size to be 
read/write in workload factors, and only consider the EC2 VM 
type in computing resource factors. As for the capacity factors, 
we can employ multiple suitable metrics in this evaluation, for 
example disk I/O latency and data throughput. However, since 
only one metric should be determined as the response in an 
experimental design [IJ, we choose the disk data throughput in 
this case. Thus, we have circled active direction, object size, 
and VM type as factors, while data throughput as response 
in the framework for designing experiments. In particular, we 
use two-level settings for the three factors: the value of active 
direction can be Write or Read; object size can be Char or 
Block; and VM type only covers MLsmall and MLlarge. In 
addition, we use "MB/s" as the unit of data throughput. 

Since only a small amount of factors are concerned, we 



can simply adopt the most straightforward design technique, 
namely Full-factorial Design |1|, for this demonstration. This 
design technique adjusts one factor at a time, which results 
in an experimental matrix comprising eight trials, as shown in 
Matrix ([T]i. For conciseness, we further assign aliases to those 
experimental factors, as listed below. Note that the sequence of 
the experimental trials has been randomized to reduce possible 
noises or biases |T| during the designing process. 

• A: Activity Direction (Write vs. Read). 

• B: Object Size (Char vs. Block). 

. C: VM Type (Ml.small vs. Ml.large). 

• Response: Data Throughput (MB/s). 
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Following the experimental matrix, we can implement 
evaluation experiments trial by trial, and fill the Response 
column with experimental results. For our convenience, here 
we directly employ the evaluation results reported in (13], as 
listed in Matrix (|2]l. 
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1 


Write 


Block 


Ml.small 


73.5 MB/s 
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Finally, different analytical techniques can be employed to 
reveal more comprehensive meanings of experimental results 
lHj for commercial Cloud services. For example, in this case, 
we can further investigate the significances of these factors to 
analyze their different influences on the disk I/O performance. 
In detail, by setting the significance level a as 0.05 |17|, 
we draw a Pareto plot to detect the factor and interaction 
effects that are important to the process of reading/writing 
data from/to EC2 disks, as shown in Figure [7] 

Given a particular significance level, Pareto plot displays a 
red reference line besides the effect values. Any effect that 
extends past the reference line is potentially important lH). In 
Figure |7] none of the factor or interaction effects is beyond the 
reference line, which implies that none of the factors or inter- 
actions significantly influences the EC2 disk I/O performance. 
Therefore, we can claim that EC2 disk I/O is statistically stable 
with respect to those three factors. However, Factor B (Data 
Size to be read/written) has relatively significant influence on 
the performance of EC2 disk I/O. Since the throughput of 
small-size data (Char) is much lower than that of large-size 
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Fig. 7. The Pareto plot of factor effects. 

data (Block), we can conclude that there is a bottleneck of 
transaction overhead when reading/writing small size of data. 
On the contrary, there is little I/O performance effect when 
switching activity directions, which means the disk I/O of EC2 
is particularly stable no matter reading or writing the same size 
of data. 

Overall, through the demonstration, we can find that this 
factor framework offers a concrete and rational foundation for 
implementing performance evaluation of commercial Cloud 
services. When evaluating Cloud services, there is no doubt 
that the techniques of experimental design and analysis can 
still be applied by using intuitively selected factors. Never- 
theless, by referring to the existing evaluation experiences, 
evaluators can conveniently identify suitable experimental 
factors while excluding the others, which essentially suggests 
a systematic rather than ad hoc decision making process. 

V. Conclusions and Future Work 

Cloud Computing has attracted tremendous amount of at- 
tention from both customers and providers in the current 
computing industry, which leads to a competitive market 
of commercial Cloud services. As a result, different Cloud 
infrastructures and services may be offered with different 
terminology, definitions, and goals |30|. On one hand, different 
Cloud providers have their own idiosyncratic characteristics 
when developing services 1*241. On the other hand, even 
the same provider can supply different Cloud services with 
comparable functionalities for different purposes. For example, 
Amazon has provided several options of storage service, such 
as EC2, EBS, and S3 |5|. Consequently, performance evalua- 
tion of candidate services would be crucial and beneficial for 
many purposes ranging from cost-benefit analysis to service 
improvement |24|. 

When it comes to performance evaluation of a computing 
system, proper experiments should be designed with respect 
to a set of factors that may influence the system's performance 
ifTSl . In the Cloud Computing domain, however, we could not 
find any performance evaluation study intentionally concern- 
ing "factors" for experimental design and analysis. On the 
contrary, most of the evaluators intuitively employed experi- 
mental factors and prepared ad hoc experiments for evaluating 



performance of commercial Cloud services. Considering factor 
identification plays a prerequisite role in experimental design, 
it is worthwhile and necessary to investigate the territory of 
experimental factors to facilitate evaluating Cloud services 
more systematically. Therefore, based on our previous work, 
we collected experimental factors that people currently took 
into account in Cloud services performance evaluation, and 
arrange them into a tree-structured framework. 

The most significant contribution of this work is that the 
framework supplies a dictionary-like approach to selecting 
experimental factors for Cloud services performance evalu- 
ation. Benefitting from the framework, evaluators can identify 
necessary factors in a concrete space instead of on the fly. In 
detail, as demonstrated in the EC2 disk I/O evaluation case in 



Section IV given a particular evaluation requirement, we can 



quickly and conveniently lookup and circle relevant factors in 
the proposed framework to design evaluation experiments, and 
further analyze the effects of the factors and their interactions 
to reveal more of the essential nature of the evaluated service. 
Note that this factor framework is supposed to supplement, 
but not replace, the expert judgement for experimental factor 
identification, which would be particularly helpful for Cloud 
services evaluation when there is a lack of a bunch of experts. 

The future work of this research will be unfolded along two 
directions. First, we will gradually collect feedback from exter- 
nal experts to supplement this factor framework. As explained 
previously. Cloud Computing is still maturing and relatively 
chaotic ll33l . it is therefore impossible to exhaustively identify 
the relevant experimental factors all at once. Through smooth 
expansion, we can make this factor framework increasingly 
suit the more general area of evaluation of Cloud Computing. 
Second, given the currently available factors, we plan to for- 
mally introduce and adapt suitable techniques of experimental 
design and analysis to evaluating commercial Cloud services. 
With experimental design and analysis techniques, this factor 
framework essentially acts as a solid base to support systematic 
implementations of Cloud services evaluation. 
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